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FIELD OF THE INVENTION 

The present invention relates to an image processing 
apparatus and method and, more particularly, to an image 
processing apparatus for processing an image containing a 
10 frame image and a method therefor. 

DESCRIPTION OF THE RELATED ART 

For example, when gradation of an image is to be 

corrected, the entire image is corrected independently of 
15 whether a frame image (to be also simply referred to as a 

"frame" hereinafter) with an impression of, e.g., a frame 

is present in the image to be processed. 

For this reason, when a white frame is contained in 

the image to be processed, highlight portions except the 
20 frame are influenced by the white color of the frame, and 

gradation is not appropriately corrected. Hence, no 

sufficient dynamic range for gradation can be obtained. 



SUMMARY OF THE INVENTION 
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The present invention has been made to solve the above 



problem, and has as its object to provide an image 
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processing apparatus capable of appropriately processing 
an image containing a frame image and a method therefor . 

In order to achieve the above object, according to 
a preferred aspect of the present invention, there is 
5 provided an image processing apparatus comprising: 

detection means for detecting an image area excluding a 
, frame image contained in an input image; generation means 
for generating correction information of the detected image 
area; and correction means for correcting the image area 
10 on the basis of the generated correction information. 

Other features and advantages of the present invention 
will be apparent from the following description taken in 
conjunction with the accompanying drawings, in which like 
reference characters designate the same or similar parts 
15 throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a view showing the functional blocks 
(modules) of software according to an embodiment of the 
20 present invention; 

Fig. 2 is a flow chart showing the operation of the 
first embodiment of the present invention; 

Fig. 3 is a view for explaining data held by a 
parameter holding section; 
25 Fig. 4 is a flow chart showing details of processing 

of a frame recognition section; 
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Figs. 5A to 5D are views for explaining the criteria 
for determining whether a pixel partially constructs a 
frames- 
Figs. 6A and 6B are views for explaining data stored 
5 in an image information holding section; 

Fig. 7 is a flow chart showing details of processing 
of an image identification section; 

Figs. 8A to 8H are views showing details of an image 
portion identification operation by the image 
10 identification section; 

Fig. 9 is a flow chart showing details of processing 
of a highlight/shadow calculation section; 

Fig. 10 is a graph showing a luminance histogram; 

Fig. 11 is a flow chart showing details of processing 
15 of a white balance calculation section; 

Fig. 12 is a flow chart showing details of processing 
of an image correction section; 

Fig. 13 is a graph showing the characteristics of a 
look-up table prepared by the image correction section; 
20 Figs. 14A and 14B are views showing an image having 

a frame with gradation; 

Fig. 15 is a flow chart showing the operation of the 
second embodiment of the present invention; 

Fig. 16 is a flow chart showing details of processing 
25 of an image identification section;. 

Figs. 17A to 17P are views for explaining an image 
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portion detection operation; 

Figs. 18A to 18L are views for explaining an image 
portion detection operation; and 

Fig. 19 is a block diagram showing the hardware 
5 arrangement of an image processing apparatus according to 
the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Preferred embodiments of the present invention will 
10 be described below in detail with reference to the 

accompanying drawings. An image processing apparatus of 
the present invention is realized by an apparatus having 
a hardware arrangement as shown in Fig. 19, for example, 
a computer apparatus such as a personal computer, or by 
15 supplying software (to be described later) to a dedicated 
computer apparatus . 

Referring to Fig. 19, a CPU 2 of a computer apparatus 
100 executes a program stored in a ROM 1 and a storage section 
8 using a RAM 3 and the storage section 8 such as a hard 
20 disk as a work memory. This program includes at least an 
operating system (OS) and software (to be described later) 
for executing processing of the present invention. 

Image data to be processed by the computer apparatus 
100 is input from an input device such as a digital still 
25 camera 7 through an input interface (I/F) 6 and processed 
by the CPU 2. The processed image data is converted by the 
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CPU 2 into a form and format according to an output device 
and sent to the output device such as a printer 11 through 
an output I/F 10. The input image data, output image data, 
and image data under processing may be stored in the storage 
5 section 8 or sent to a monitor 5 such as a CRT or an LCD 
through a video I/F 4 to display the image, as needed. These 
processing and operations are instructed by the user 
through a keyboard as an input device or a mouse as a pointing 
device connected to a keyboard I/F 9. 
10 As the input and output I/Fs 6 and 10, SCSI or GPIB 

interfaces as general-purpose interfaces, parallel 
interfaces such as Centronics, and serial interfaces such 
as RS232, RS422, IEEE1394, or USB (Universal Serial Bus) 
are used. 

15 As the storage section 8, not only a hard disk but 

also a storage medium such as a magneto-optical disk (MO) 
or an optical disk including a digital video disk (DVD-RAM) 
can be used. As the device for inputting image data, a 
digital video camera, an image scanner, or a film scanner 

20 can be used in addition to the digital still camera. Image 
data can also be input from the above storage medium or 
through a communication medium. As the device for 
outputting image data, a printer such as a laser beam 
printer, an ink-jet printer, or a thermal printer, or a film 

25 recorder can be used. Processed image data may be stored 
in the above storage medium or sent to a communication 
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medium. 

First Embodiment 

Fig. 1 is a view showing the functional blocks 
(modules) of software of the first embodiment. Fig. 2 is 
5 a flow chart showing the operation of the first embodiment. 
The operation of this embodiment will be described below 
in detail in units of functional blocks. 
[ Frame Recognition] 

In step SI, an input image 1 is read by an image input 

10 section 2 and stored in an image buffer 4. In step S2, the 
image data buffered in the image buffer 4 is checked in units 
of pixels by a frame recognition section 8, whose processing 
is shown in Fig. 4 in detail. It is determined whether a 
pixel partially constructs a frame (step S41) , and the 

15 determination result is stored in an image information 
holding section 9 (step S42) . On the basis of determination 
in step S43, processing in steps S41 and S42 is repeated 
for all image data buffered in the image buffer 4, and then, 
the flow advances to step S3. 

20 Determination in step S41 is done by comparing the 

color of a pixel of interest with the colors of eight pixels 
(adjacent pixels) adjacent to the pixel of interest. If 
a condition for recognizing a frame is satisfied, the pixel 
of interest is marked as part of a frame. If the condition 

25 is not satisfied, the pixel of interest is marked not to 
construct a frame. 
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Figs. 5A to 5D are views for explaining the criteria 
for determining whether a pixel partially constructs a 
frame. When any one of the following conditions is 
satisfied, a pixel e of interest is recognized as part of 
5 a frame . 

(1) As shown in Fig. 5A, pixels ^, b, d, and e have 
the same color. 

(2) As shown in Fig. 5B, pixels b, c, e, and f have 
the same color. 

10 (3) As shown in Fig. 5C, pixels e, f, h, and i have 

the same color. 

(4) As shown in Fig. 5D, pixels d, e, g, and h have 

the same color. 

The "same color" in the above conditions may be 
15 replaced with, e.g., "colors within a predetermined range" . 

Figs. 6A and 6B are views for explaining data stored 

in the image information holding section 9. The image 

information holding section 9 holds data of 1 bit/pixel 

(Fig. 6B) in correspondence with image data of 8 bits/pixel 
20 of each color (Fig. 6A) . That is, the image information 

holding section 9 holds binary data with the same sizes as 

the vertical and horizontal sizes of an image stored in the 

image buffer 4 . 

[ Image Identification] 
25 In step S3, an image portion, i.e., an image portion 

other than the frame is identified from the data stored in 
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the image information holding section 9 by an image 
identification section 11, whose processing is shown in 
Fig. 7 in detail (steps S51 to S54) . Information of the 
upper, lower, left, and right ends of the image portion as 
5 the identification result is stored in a parameter holding 
section 5. Terms "upper end", "lower end", "left end", and 
"right end" will be described later in detail. Detection 
of an image portion except the frame will be described below . 
However, the frame portion can also be detected in 

10 accordance with almost the same procedure. 

Figs. 8A to 8H are views showing details of an image 
portion identification operation by the image 
identification section 11. In step S51, the left end of 
the image is detected. To do this, the image is checked 

15 in units of columns from the left to the right . The position 
of the first column containing a pixel marked not to 
construct a frame is detected as the left end (Figs. 8A and 
SB) . 

In step S52, the upper end of the image is detected. 
20 The image is checked in units of rows from the upper side 
to the lower side. The position of the first row containing 
a pixel marked not to construct a frame is detected as the 
upper end (Figs, 8C and 8D) . 

In step S3, the right end of the image is detected. 
25 The image is checked in units of columns from the detected 
left end to the right. The position of a column on the 
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immediately left side of a column in which all pixels are 

marked to construct a frame is detected as the right end. 

When the right end is not detected, i.e. , a column in which 

all pixels are marked to construct a frame is not detected 
5 until the right end of the image, the position of the 

rightmost column of the image is set as the right end 

(Figs . 8E and 8F) . 

In step S4, the lower end of the image is detected. 

The image is checked in units of rows from the detected upper 
10 end to the lower side. The position of a row on the 

immediately upper side of a row in which all pixels are 

marked to construct a frame is detected as the lower end. 

When the lower end is not detected, i.e., a row in which 

all pixels are marked to construct a frame is not detected 
15 until the lower end of the image, the position of the 

lowermost row of the image is set as the lower end (Figs. 8G 

and 8H) . 

In the above description, a column or row having a 
pixel marked not to construct a frame or a column or row 

20 in which all pixels are marked to construct a frame is 
detected. However, in consideration of a case wherein an 
end of the frame tilts, curves, or undulates,; a column or 
row having at least a predetermined number of pixels marked 
to construct a frame or at least a predetermined number of 

25 consecutive pixels marked to construct a frame may be 
detected. 
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[Calculation of Highlight Point and Shadow Point] 

In step S4, a highlight point and a shadow point are 
calculated by a highlight /shadow calculation section 6, 
whose processing is shown in Fig. 9 in detail, on the basis 
5 of the information stored in the parameter holding section 
5, and stored in the parameter holding section 5, More 
specifically, in step Sll, image data of the image portion 
except the frame is read out from the image buffer 4, and 
a luminance histogram shown in Fig. 10 is generated. Next, 

10 on the basis of the generated histogram, a highlight point 
LH and a shadow point LS are calculated in steps S12 and 
S13. The highlight point LH is the minimum luminance value 
in the highlight area. The shadow point LS is the maximum 
luminance value in the shadow area. 

15 In the luminance histogram shown in Fig. 10, since 

luminances in the highlight area (99% to 100%) are 230 to 
255, the highlight point LH is 230. Additionally, since 
luminances in the shadow area (0% to 1%) are 0 to 14, the 
shadow point LS is 14. 

20 [Calculation of White Balances] 

In step S5, the white balances and black balances are 
calculated by a white balance calculation section 7, whose 
processing is shown in Fig. 11 in detail, on the basis of 
the information stored in the parameter holding section 5, 

25 and stored in the parameter holding section 5. More 

specifically, in steps S21 and S22, each pixel is read out 
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from the image buffer 4, the average luminance value (white 
balance) of pixels with luminances falling between the 
highlight point LH and a corrected highlight point HP is 
calculated for each of the R, G, and B colors, and the average 
5 luminance value (black balance) of pixels with luminances 
falling between a corrected shadow point SP and the shadow 
point LS is calculated for each of the R, G, and B colors. 

Referring to Fig. 10, the average luminance of pixels 
with luminances falling within the range from LH = 230 to' 

10 HP = 245 is calculated as the white balance for each of the 
R, G, and B colors, and the average luminance of pixels with 
luminances falling within the range from SP = 10 to LS = 
14 is calculated as the black balance for each of the R, 
G, and B colors. These results are stored in corresponding 

15 registers RH, GH, BH, RS, GS, and BS in the parameter holding 
section 5 (Fig. 3) . 
[ Image Correction] 

In step S6, gradation of the image is corrected by 
an image correction section 10, whose processing is shown 

20 in Fig.. 12 in detail, on the basis of the information stored 
in the parameter holding section 5, and the correction 
result is written in the image buffer 4 . More specifically, 
a look-up table for gradation correction is prepared on the 
basis of the white balances and black balances stored in 

25 the parameter holding section 5 (stepS31). Image data read 
out from the image buffer 4 in units of pixels is subjected 
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to gradation correction using the look-up table. The 
corrected image data are written in the image buffer 4 (step 
S32) . 

Fig. 13 is a graph showing the characteristics of the 
5 look-up table. The look-up table is prepared on the basis 
of the white balances RH, GH, BH, and white point LH, and 
the black balances RS, GS, and BS, and black point LS, In 
the example shown in Fig. 13, the gamma correction level 
for the highlight portion increases in the order of green, 
10 blue, and red. In this way, by emphasizing green and blue 
with respect to red, so-called color fog of a bluish 
(fogged with blue) image can be corrected. 
[Image Output] 

Finally, in step S7, the image which has undergone 
15 gradation correction and buffered in the image buffer 4 is 
output by an image output section 3 as an output image 12. 
[Parameter Holding Section] 

Fig. 3 is a view showing data held by the parameter 
holding section 5. In the initial state, appropriate 
20 values are stored as the corrected highlight point HP and 
corrected shadow point SP. 
[Recognition of Frame with Gradation] 

When the following conditions are set for 
determination in step S41 in consideration of a frame with 
25 gradation as shown in Fig. 14A, the pixel e of interest can 
be recognized to construct a frame (Fig. 14B) . To 



- 12 - 



determine the following conditions, RGB image data is 
temporarily converted into HSB data or HSL data. This 
conversion technique is known and a detailed description 
thereof will be omitted. 



same hue, and the difference between the lightness and 
saturation has a predetermined value or less. 

(2) Pixels b, c, e, and f shown in Fig. 53 have the 
same hue, and the difference between the lightness and 

10 saturation has a predetermined value or less. 

(3) Pixels e, f, h, and i shown in Fig. 5C have the 
same hue, and the difference between the lightness and 
saturation has a predetermined value or less. 

(4) Pixels d, e, g, and h shown in Fig. 5D have the 
15 same hue, and the difference between the lightness and 

saturation has a predetermined value or less. 
Second Embodiment 

In the first embodiment, gradation correction when 
one image portion (e.g., a photograph) is contained in one 

20 image has been described. However, when frame recognition 
of the present invention is applied, even when a plurality 
of image portions are contained in one image, gradation 
correction can be appropriately performed for each image 
portion. The second embodiment in which, for example, two 

25 image portions are recognized, and gradation correction is 
performed for each of the two recognized image portions will 
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(1) Pixels b, d, and e shown 



in Fig. 5A have the 
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be described below. An image portion detection method to 
be described below can be applied to detect not only two 
image portions but also three or more image portions, as 
a matter of course. 
5 Fig. 15 is a flow chart showing the operation of the 

second embodiment. The operation of the second embodiment 
will be described below in detail in units of functional 
blocks . 

[Frame Recognition] 

10 In step S.61, an input image 1 is read by an image input 

section 2 and stored in an image buffer 4. In step S62, 
the image data buffered in the image buffer 4 is checked 
in units of pixels by a frame recognition section 8. It 
is determined whether a pixel partially constructs a frame 

15 (step 541), and the determination result is stored in an 
image information holding section 9 (step S42) . On the 
basis of determination in step S43, processing in steps S41 
and S42 is repeated for all image data buffered in the image 
buffer 4, and then, the flow advances to step S63. 

20 [Image Identification] 

In step S63, an image portion, i.e. , an image portion 
other than the frame is identified from the data stored in 
the image information holding section 9 by an image 
identification section 11, whose processing is shown in 

25 Fig. 16 in detail (steps S71 to S76) . Information of the 
upper, lower, left, and right ends of the image portion as 
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the identification result is stored in a parameter holding 
section 5. 

The operation of the image identification section 11 
will be described in detail. In step S71, the left end of 
5 the image is detected- To do this, the image is checked 
in units of columns from the left. The position of a column 
containing a pixel marked not to construct a frame is 
detected as the left end. Subsequently, in step 372, it 
is determined whether the left end is detected. If NO in 

10 step S72, detection is ended. If YES in step S72, the flow 
advances to step 373. 

In step 873, the upper end of the image is detected. 
The image is checked in units of rows from a row containing 
a pixel marked not to construct a frame and located at the 

15 uppermost portion of the column at the left end detected 
in step S71 to the upper side. A row having at least a 
predetermined number of consecutive pixels marked to 
construct a frame is detected. The position of a low 
immediately below the row is detected as the upper end. 

20 In step S74, the values of the detected left and upper 

ends are set as the initial values of the right and lower 
ends of the image. In step S75, the right end of the image 
is detected. The image is checked from the position of the 
right end initially set in step S74 to the right in units 

25 of columns . A column having at least a predetermined number 
of consecutive pixels marked to construct a frame is 
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detected. The position of a column immediately on the left 
side of the column is detected as the right end. 

In step S76, the position of the right end of the image 
is compared with that of the lower end. Processing advances 
5 on the basis of the comparison result. 

(1) When the right end is on the lower left side of 
the lower end, processing is ended. 

(2) When the right end is on the upper side of the 
lower end, the flow returns to step S75- 

10 (3) When the lower end is on the left side of the 

right end, the flow advances to step S77. 

In step S77, the lower end of the image is detected. 
The image is checked from the current lower end position 
to the lower side in units of rows. A row having at least 

15 a predetermined number of consecutive pixels marked to 
construct a frame is detected. The position of a row 
immediately above the row is detected as the lower end. 

When detection processing shown in Fig. 16 is ended, 
it is determined in step S64 whether the upper, lower, left, 

20 and right ends of the image are detected, i.e., an image 
portion is detected- If YES in step S64, information 
representing the upper, lower, left, and right ends of the 
image portion are stored in the parameter holding section 
5, and the flow advances to step S65. If NO in step S64, 

25 i.e., when detection is ended, the flow advances to step 
S69, and the image which has undergone gradation correction 
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and buffered in the image buffer 4 is output by an image 
output section 3 as an output image 12. 

Steps S65 to S67 correspond to steps S4 to S6 in Fig. 2 
and have substantially the same processing contents as 
5 described above, and a detailed description thereof will 
be omitted. 

In step S68, information in the area of an image 
information holding section 9, which corresponds to the 
image portion which has undergone gradation correction, is 

10 marked again to construct a frame. After the information 
in the image information holding section 9 is updated, the 
flow returns to step S63 to detect the next image portion. 
[Example 1 of Image Recognition] 

Figs. 17A to 17P are views for explaining image 

15 recognition when one image contains two image portions. 

At the time point of step S62, information as shown 
in Fig. 17A is stored in the image information holding 
section 9. Next, in step S71, a column containing a pixel 
determined not to construct a frame is searched for from 

20 the left in units of columns to detect the left end of the 
image (Fig. 17B) . In step S73, on the right side of the 
detected left end, a row having at least a predetermined 
number of consecutive pixels marked to construct a frame 
is searched for to the upper side in units of rows to detect 

25 the upper end of the image (Figs. 17C and 17D) . In step 
S74, the same values as those of the left and upper ends 
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are set as the initial values of the right and lower ends 
of the image. 

In step S75, the right end of the image is detected. 
The image is checked from the position of the currently set 
5 right end of the image to the right in units of columns. 
The position of a column immediately on the left side of 
a column having at least a predetermined number of pixels 
marked to construct a frame is detected as the right end 
(Figs. 17E and 17F) . 

10 In step S76, the position of the right end is compared 

with that of the lower end. In the example shown in 
. Figs. 17A to 17P, since the lower end is on the left side 
of the right end, the flow advances to step S77: In step 
S77, the lower end of the image is detected. The image is 

15 checked from the currently set lower end to the lower side 
in units of rows. The position of a row immediately above 
a row having at least a predetermined number of consecutive 
pixels marked to construct a frame is detected as the lower 
end (Figs. 17G and 17H) . 

20 In step S76, again, the position of the right end is 

compared with that of the lower end. In this case, since 
the right end is on the lower left side of the lower end, 
the area of an image portion is determined, and the flow 
advances to step S64. Since the image portion is detected, 

25 steps S65 to S67 are executed on the basis of determination 
in step S64. The detected image portion is subjected to 
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gradation correction. In step S68, the information in the 
image information holding section 9 is updated, and pixels 
corresponding to an area indicated by a broken line in 
Fig. 171, i.e., the image portion which has undergone 
5 gradation correction, are marked again to construct a 
frame . 

In step S63, again, another image portion is detected 
in accordance with the same procedure as described above 
p (Figs. 171 to 17P) . Since the image portion is detected, 

'5:s 

y 10 steps S65 to S67 are executed on the basis of determination 

iji - in step S64. The image portion is subjected to gradation 



correction. In step S68, the information in the image 
information holding section 9 is updated- After this, the 
flow returns to step S63 again. However, since only areas 

15 marked to construct frames are stored in the image 

information holding section 9, detection is ended on the 
basis of determination in step S72- After determination 
in step S64, an image which has undergone gradation 
correction is output in step S69. 

20 [Example 2 of Frame Recognition] 

Figs. 18A to 18L are views for explaining image 
recognition. In Figs. 18A to 18L, an image portion 
represented by data stored in the image information holding 
section 9 after execution of step S62 has a U shape due to 

25 some reason. The original image portion has, e.g., a 
rectangular photographic image. 
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First, in step S71, the left end of the image is 
detected (Figs. 18A and 18B) . In step S73, the upper end 
of the image is detected (Figs. 18C and 18D) . In step S75, 
the right end of the image is detected (Figs. 18E and 18F) . 
5 In step S76, the position of the right end is compared with 
that of the lower end. Since the lower end is on the left 
side of the right end, the flow advances to step S77. 

In step S77, the lower end of the image is detected 
(Figs. 18G and 18H) • In step S76, again, the position of 
10 the right end and that of the lower end are compared. Since 
the right end is on the upper side of the lower end, the 
flow returns to step S75. 

In step S75, the right end of the image is detected 
(Figs. 18 J and 18K) . In step S76, again, the position of 
15 the right end is compared with that of the lower end. Since 
the right end is on the lower left side of the lower end, 
the image portion is determined, and the flow advances to 
step S64. 

As described above, according to the above-described 
20 embodiments, since an image containing a frame image is 
subjected to gradation correction excluding the frame image, 
the gradation can be appropriately corrected without any 
influence of the color or luminance of the frame image. In 
addition, a frame image with gradation can also be 
25 recognized using a similar algorithm. Furthermore, with 
application of this algorithm, even when an image contains 
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a plurality of images such as photographs separated by frame 
images, appropriate gradation correction can be performed 
for each image. 

[Terms: Left End, Upper End, Right End, and Lower End] 
5 In the above description of embodiments, the left, 

upper, right, and lower ends of an image are detected. More 
. exactly, the coordinates of positions indicated by hollow 
bullets in Figs. 17A to 18H are detected, and these 
positions are compared with each other. For example, a 
10 right end or lower end means the coordinates of a position 
where a line corresponding to the right end of an image 
portion crosses the contour of the entire image or the 
contour of the image portion. 

As has been described below, according to the present 
15 invention, an image containing a frame image can be 
appropriately processed. 

As many apparently widely different embodiments of the 
present invention can be made without departing from the 
spirit and scope thereof, it is to be understood that the 
20 invention is not limited to the specific embodiments thereof 
except as defined in the appended claims. 
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